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Overview and Scope 


Behavioral tasks are an increasingly important component in neurophysiology studies, and are 
often considerably more complex than passive sensory studies. The description of behavior can 
be divided into stimulus sensory modality, behavioral contingencies, and behavioral outputs. In 
the current NWB format specification (release 2.2.5, May 2020), behavioral outputs can be 
stored as events (lick, nose poke), and the stages of a task (e.g. waiting period, cue period, 
reward period) may be stored as epochs. However there is currently no way to store 
descriptions of control flow and behavioral contingencies that provide critical context for how 
trial events and behavioral outputs produce subsequent actions within a trial. The scope of this 
proposal is to define an NWB extension to enable scientists to systematically describe 
behavioral tasks as part of a neurophysiology experiment. 


Aims 

e Use a standard markup language (BEADL) to build state machine descriptions of task 
logic and control flow within a trial and across trials in a session 
Define a set of essential metadata for common tasks 
Build a controlled vocabulary of key terms for behavioral tasks and construct a domain 
ontology 
Write a new schema extension for behavioral tasks in NWB 
Build software tools for visualization, construction, and emulation of tasks from these 
machine-readable descriptions 


State machines for task modeling 


State machines are ‘A model of computation consisting of a set of states, a start state, an input 
alphabet (inputs the SM responds to), and a transition function that maps input symbols and 
current states to a next state.’ (NIST). Typical uses are e.g. in software engineering, where state 
machines can be used to define the behavior of components in a GUI. State charts are a subset 
of state machines that allows hierarchical states and parallel processing within a state. 

e Original Harel state chart paper: Statecharts: a visual formalism for complex systems 

e Statechart tutorial: https://statecharts.github.io/ 





State machines are already used in some popular software packages for design and execution 
of behavioral experiments (Bpod, B-control, Ratrix). State machines are used to define the logic 
of single trials and the handling of signals and events within a trial. For example, Bpod was 
developed in Kepecs lab for use in control of behavioral experiments: “Bpod builds on the 
central design concept of B-control, a system provided by Brody Lab at Princeton University for 


rodent behavior measurement. Experimental trials are constructed in MATLAB as finite state 
machines, and executed on a separate real-time Linux computer.” 
e State matrices in Bpod: 
https://sites.google.com/site/bpoddocumentation/bpod-user-quide/using-state-matrices 
e Bpod Matlab protocol example: 
https://sites.google.com/site/bpoddocumentation/bpod-user-quide/protocol-writing 
e B-control state machine definition: 


https://brodylabwiki.princeton.edu/bcontrol/index.php?title=Real-Time Linux State Mac 
hine 


BEADL 


BEADL is a specification language for behavioral tasks that is currently being developed by 
current and former members of the Kepecs lab. The goal is to have a specification format that is 
machine-readable and executable (for testing/validation), with an included tool for building and 
visualizing task structure and control flow in a GUI. 

e BEADL Specification Document 


Out of scope 


e Other species: 

o Non-human primates - We are focused on rodent task models (see use cases), 
however many behavioral studies involve NHPs and archives will include primate 
data. NWB should be able to accommodate these types of experiments, and 
there may be more diversity/complexity in primate tasks than those used in 
rodents. 

o Humans - Since NWB is an electrophysiology standard we probably won't 
handle much human data. However task structure and logic can have parallels 
with animal studies, and neuroimaging data models (e.g., NIDM) will have 
terminology and even task structure that differ from that used in animal studies. 
There are other schemas/ontologies for human tasks (see below), can we/do we 
want to keep some consistency? 

e Virtual/natural environments: Referring to experiments where animals are allowed to 
explore an environment naturalistically, i.e. not a ‘controlled’ environment. Behavioral 
response may be measured from continuous pose or movement data as analyzed 
through e.g. DeepLabCut or other software for analyzing and classifying movement data. 
It is unclear how a State Machine description would apply to these studies (# of stimuli - 
inf, # of responses = inf?) 

e The 80/20 Rule 


Definitions 


The terminology that will be used includes the following: 


Task: 

State machine: 

Stimulus: 

Response: 

Reward: 

Instruction: 

Trial: 

Decision variable/decision rule: 
Lapse trial: 

Guess: 


Task metadata 


Beyond the state machine for task logic/flow, we need fields for storing task metadata that is 
essential for data analysis. 
e Stimulus metadata 
o Modality 
o Variable parameter values (for e.g. computing decision variables - contrast in 
contrast discrimination task) 
o Duration/timing of stimulus presentation 
e Reward metadata 
o Modality 
o Value (amount rewarded) 
e Signals/events that are not task-aligned 
o Motor output data (wheel running, wheel turning) 
o Pose data (from e.g. DeepLabCut) 


Related Schemas and Data Models 


CogPO (http://www. wiki.cogpo.org/index.php?title=Main_ Page) - human 
Neurolmaging Data Model NIDM (https://scicrunch.org/nidm-terms; 
httos://github.com/incf-nidash/nidm-terms) - also human 

e HBP openMINDS (https://github.com/HumanBrainProject/openMINDS) - just getting 
started 

e odML (hittps://github.com/G-Node/odm|-terminologies) - for animal studies, but also just 
getting started 


e Datajoint’s IBL schemas: https://github.com/int-brain-lab/IBL-pipeline 


Use Cases 


UC1: Allen Visual Behavior - Change Detection 


This is a continuously-presented, flashed natural scene change detection task. For each trial, a 
natural image is presented for 250ms followed by 500ms of gray screen. On GO trials, a change 
in image identity occurs and mice must lick within the 750ms (one trial sequence) response 
window to receive a water reward. On CATCH trials, no stimulus change occurs (same image 
presented again) and the behavioral response is measured to quantify guessing behavior. To 
test whether expectation signals were present in the visual cortex during this task, ~5% of all 
non-change flashes were omitted. 


Per session metadata: 

duration (~1 hr) 

image set (1 of 4) 

response window - time for reward availability 

timeout duration - extra time punishment for early responses 
min_no_lick_time - time with no licks to mark end of trial 
catch frequency - frequency of trials where no change occurs 


e@ wee 
Per trial metadata: 
e change type (GO, CATCH) 
e change time (time/frame number) 
e next_image (image identity [0..7]) 


Articles: 
o Experience shapes activity dynamics and stimulus coding of VIP inhibitory cells 
o Training model: Quantifying and Modelling Transfer Learning in Mice Between 
Consecutive Training Stages of a Change Detection Task 


UC2: Kepecs Lab - Confidence Reporting Task 


This is a 2AFC task with a behavioral report of confidence. Rats poke in a center port and 
receive an auditory (stereo click train) stimulus. The difficulty of auditory decisions is varied by 
adjusting the balance of left/right rates in binaural streams of random clicks. Rats express their 
choice about stimulus identity by entering one of two side ports. On each trial, the reward is 
delayed by a random amount of time that is sampled on each trial from a truncated exponential 
distribution. Once rats have entered the choice port they have to wait until a reward is delivered 
or decide to give up and restart a new trial. In 10% of the trials, reward is withheld to measure 
how long the rats would have been willing to wait on correct trials. 


Article: 
e Behavior- and Modality-General Representation of Confidence in Orbitofrontal Cortex 


Per trial metadata: 
e trial type (reward given or withheld) 
stimulus delay - time animal must wait at home port before stimulus 
correct delay - time animal must wait for reward (0.6-8s, mean 1.5s) 
sampling time - the amount of time the animal sampled the stimulus 
movement time - the amount of time the animal spent in moving from the center port to 
the choice port 
e time investment - the amount of time the animal invested without getting a reward 


UC3: IBL - 2AFC/AUC and Contrast Discrimination Tasks 


In the IBL protocol, head-fixed mice are trained to determine the location of a vertical grating on 
the screen and report their judgment by moving a wheel left or right, which moves the grating to 
the center of the screen. Grating contrast is varied to adjust task difficulty. There is an unforced 
choice variant of this task in which no grating is shown and the animal is rewarded for not 
responding. In addition there is a contrast discrimination variant where gratings are presented 
on both sides of the screen, and the mouse is rewarded if the higher contrast grating is moved 
to the center of the screen. 


Per session metadata: 

e duration (varies depending on task performance, 90min cutoff) 

e reward volume (less reward used as mouse was better trained) 
Per trial metadata: 

e stimulus location (L/R) 

e stimulus contrast (variable) 

e  inter-trial interval (variable, after stimulus offset) 

e pre-stimulus quiescent period duration (variable, can be zero) 


Articles: 
e Pipeline paper: Standardized and reproducible measurement of decision-making in mice 


e Methods paper: High-Yield Methods for Accurate Two-Alternative Visual Psychophysics 
in Head-Fixed Mice 


Specs for control hardware: https://github.com/harp-tech/IBL_ behavior control 


Additional Use Cases 


- Frank Lab (n-AFC maze task designs): 


- Preprint: https:/Awww.biorxiv.org/content/10.1101/2020.12.14.420174v1 
- Reach-to-grasp task from Neo/Elephant paper 


- Handling Metadata in a Neurophysiology Laborato Frontiers in 
Neuroinformatics 
- Interval tasks (2IFC, 2IUC) 


Other Resources 


UML and scXML (state chart XML) for state machine definition: These are existing 
standards for defining state machines, and are the formats used in many commonly-used 
software tools for working with state machines (MagicDraw, Yakindu). 
e W3C: State Chart XML (SCXML): State Machine Notation for Control Abstraction 
e Apache Commons (warning - dead links) - SCXML- Commons SCXML Usage - Five 
minute SCXML tutorial 
e From Yakindu (state machine software tool) documentation: 
https://www.itemis.com/en/yakindu/state-machine/documentation/user-quide/scxml_integ 
ration 
‘The XML defines all the states, transitions, events and variables used in the state 
machine. Besides the state machine’s structural elements, the SCXML standard also 
defines the execution semantics of that state machine. And this is one of the biggest 
advantages of SCXML — the same statechart model can run on different SCXML engines 
on different platforms and it always behaves in exactly the same way — as long as the 
engines conform to the specified execution semantics.’ 


INCF SIG on neuroscience data structure: (Repository: INCF/neuroscience-data-structure: 
Space for discussion of a standardized structure (directory layout + metadata) for experimental 
data in systems neuroscience, similar to the idea of BIDS in neuroimaging) 

This working group is aiming to coordinate efforts across multiple projects for data sharing 
(NWB, DANDI, eBrains, HBP). This includes associated projects for defining metadata terms, 
openMINDS and odML. 


Existing NWB Components 


At present, NWB has no dedicated fields for holding descriptions of behavioral tasks and 
metadata, outside of general fields for storing experimental descriptions/metadata. NWB does 
have components for storing behavioral epoch/event data: 


e BehavioralEpochs: An interface for an IntervalSeries with timestamps delimiting epochs 
in the recording session. Epochs can be regularly spaced (rate) or irregularly spaced 
(individual timestamps). 

e BehavioralEvents: BehavioralEvents is used for storing irregular events, like licks or 
nose pokes. 

e BehavioralTimeSeries: BehavioralTimeSeries is for storing continuous behavioral data, 
e.g. wheel position in the IBL tasks. 


e SpatialSeries: are used to store position or direction relative to a reference frame. 
Useful for e.g. maze tasks. EyeTracking, CompassDirection, and Position data 
interfaces store SpatialSeries data with the relevant reference frame (text) 


Published Datasets 


e AIBS Visual Behavior 
o Datasets: 
https://figshare.com/collections/Experience_shapes_activity dynamics _and_stim 
ulus coding of VIP inhibitory cells/4858779/1 


e IBL/Steinmetz Lab 
o IBLDANDIset: httos://dandiarchive.org/dandiset/000045/draft 
o IBL Figshare: A standardized and reproducible method to measure 
decision-making in mice: Data 
o IBLArchitecture paper: 
https://www.biorxiv.org/content/10.1101/827873v3.full.odf+html 
o Steinmetz paper NWB data: https://figshare.com/articles/steinmetz/9974357 


e DANDI archive examples/Buffalo Lab/others .... (?Links) 


Work Plan 


Jobs 
e Controlled vocabulary and ontology definition 
e Define list of essential/minimal metadata for tasks 
e Build schema 
e Data scientists to validate schema with example data 
e Make extension, tutorial and outreach 


NWB Extension 


Open Questions 


e Is BEADL flexible enough to handle arbitrary tasks eg outside of Bpod/Bcontrol 
descriptions? 

e Why not use DynamicTables for BehavioralEpochs in NWB (like is used for Epochs)? 
Would allow tags for labelling. 


Topics and Issues to Address 


Agenda and Discussion Notes (12/18/2020 - AIBS/Kepecs/NWB) 


Agenda Items: 
1) Introductions (5 minutes) 
2) Overview of state machines for task modeling and goals (Pam - 20 minutes) 
3) Overview of BEADL project and goals (Michael, Marion - 10 minutes) 
4) Update on BEADL/NWB integration and goals (Michael, NWB team - 5-10 minutes) 
5) Discussion of issues that arise, refine aims and work plan, set priorities for next steps 
(15 minutes) 


- Some considerations for ‘minimal’ or ‘essential’ metadata (Pam): 

e What metadata would be required to replicate the experiment elsewhere? 

e Are metadata provided which may explain variability (e.g., between subjects or 
recordings) in the recorded data? 

e Are metadata provided which enable access to subsets of data to address 
specific scientific questions? 

e ls the recorded data described in sufficient detail, such that an external 
collaborator could understand them? 


